https://www.linkedin.com/pulse/criteria-good-contribution-hard-steve-powell-pf72e/

I just read Julian King's response to a blog by Drew Koleros and Michael Moses. Drew and Michael were saying, these complex evaluations are so hard; we have to go beyond simply measuring against pre-determined criteria, partly because everything changes. Julian responded by saying that to evaluate the value or worth of something, we can and should not just measure but establish criteria. Even in complex situations establishing then applying criteria isn't so hard, if you pick the right kind of criteria (not too specific and not too vague). I think he's arguing also that to be a genuine assessment of value, the criteria should be established in advance - a key element of accountability which addresses the summative purpose of evaluation. I think it is this which Drew and Michael are saying is so hard.

Here is the first part of Julian's example criterion for a public transport initiative aimed at nudging residents’ attitudes to cars and driving.

The transport system demonstrates a meaningful shift toward diverse, accessible, and sustainable mobility modes, reducing car dependency while increasing uptake and satisfaction with public and active transport, including rapid transit, walking and cycling. The optimal mix and sequencing of transport options is arrived at through genuine collaboration with stakeholders, and informed by sound analysis provided by transport experts.

All good. But this example is all about assessing progress in outcome areas, wherever that comes from. The really hard part in evaluation is assessing a project's contribution to progress, which is (I think) what Drew and Michael are talking about. The criterion has to cover both the outcome and the contribution at the same time. That's the part that's hard.

You'd think Thomas Aston's contribution rubrics would solve this problem: together with stakeholders, set criteria for:

• the significance of the outcome,

• the level of the contribution

• and the strength of the evidence,

.... and then do the assessment and combine the three scores. That's also great but it's designed for assessing contribution post-hoc, as in Outcome Harvesting so it doesn't answer Julian's implicit accountability plea to establish criteria in advance.

So an evaluation criterion, in-advance, would have to look something like this:

The initiative will show a meaningful and well-evidenced contribution to the transport system taking a meaningful shift toward diverse, accessible, and sustainable mobility modes...

That's not very elegant.

Here's where evaluation criteria and progress criteria diverge. You could put Julian's example above the door of the initiative's office to remind staff what the aim is. Like a Vison in Outcome Mapping. I wouldn't the evaluation criterion above anyone's door.https://www.linkedin.com/pulse/criteria-good-contribution-hard-steve-powell-pf72e/